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Purpose of Declaration 

1 . This declaration is to establish conception of the invention in this application at a date at 
least prior to August 21, 2003, the effective date of the cited patent application publication: 

US2005/0044487 to Bellegarda et al. (application no. 10/644,815). 

2. The person making this declaration is one of the joint inventors and is the inventor of the 
subject matter of the rejected claims. (MPEP 715.04). 

Facts and Documentary Evidence 

3. To establish the date of conception and diligence from a date prior to the effective date of 
the reference until the date of filing of the invention of this application, the following attached 
documents are submitted as evidence: 

• FXPAL Internal Invention Proposal signed and witnessed on a date prior to 
August 21, 2003) 
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• Email Correspondence regarding preparation of the application between the 
drafting attorney and the primary inventor (August 7, 2003) 

• Email Correspondence regarding a convenient date for inventor interview 
between the primary inventor and drafting attorney (August 12, 2003) 

• Further Email Correspondence regarding a convenient date for inventor interview 
between the primary inventor and drafting attorney (August 21, 2003) 

• Email Correspondence regarding a first draft of the application between the 
primary inventor and drafting attorney (October 14, 2003) 

• Email Correspondence conveying the first draft of the application between the 
primary inventor and drafting attorney (October 30, 2003) 

• Email Correspondence conveying inventor's comments regarding the first draft of 
the application between the primary inventor and drafting attorney (October 31, 
2003) 

• Email Correspondence conveying references to the drafting attorney between the 
primary inventor and drafting attorney (November 7, 2003) 

• Email Correspondence conveying another draft to the primary inventor between 
the primary inventor and drafting attorney (November 13, 2003) 

• Email Correspondence verifying inventorship information between the primary 
inventor and drafting attorney (November 24, 2003) 

• Email Correspondence conveying a final draft to the primary inventor between 
the primary inventor and drafting attorney (December 2, 2003) 
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4. The attached FXPAL Internal Invention Proposal, has been signed and witnessed on a 
date prior to August 21, 2003. Thus, the invention in this application has been conceived at least 
prior to August 21, 2003 which is the effective date of the reference. 

Diligence 

5. The attached documents also establish diligence beginning from a date prior to the 
effective date of the reference on August 21, 2003, until the date of filing of the application on 
December 9, 2003. (MPEP 715.07(a)). 

Time of Presentation of the Declaration 

6. This declaration is submitted after a final office action together with a Request for 
Continued Examination (RCE). Therefore, the declaration is timely. (MPEP 715.09). 

Declaration 

As a person signing below, I hereby declare as follows: 

I am a co-inventor of the subject matter claimed in the present patent application. At the 
time of inventing the subject matter of the present patent application, I was an employee of Fuji 
Xerox Co. LTD. 

On or about a date at least prior to August 21, 2003, I provided the enclosed invention 
disclosure, directed to the subject matter of the present patent application, to a witness whose 
signature appears on the disclosure stating that he has read and understood the disclosure. 

To my personal knowledge, the copy of the invention disclosure attached to this 
Declaration is true and correct copy of the invention disclosure that I submitted on or about a 
date at least prior to August 21, 2003. The copy of the invention disclosure attached to this 
Declaration has been kept in business records of the law firm in ordinary course of business. 

The enclosed invention disclosure establishes that I was in possession of the invention at 
least prior to August 21, 2003. 
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Between the date of submission of the invention disclosure and the date of filing of the 
application on December 9, 2003, I or one of my co-inventors was in contact with the patent 
attorney drafting the application either directly or through FXPAL patent counsel and diligently 
reviewed the drafts of the application provided by the drafting attorney. 

The enclosed evidence of the communication (10 emails) between the drafting patent 
attorney and myself, or my co-inventor, establish that I exercised diligence from at least a date 
prior to the effective date of the reference to the date of filing of the instant application. 

The enclosed invention disclosure was created by me and my co-inventors and the draft 
application submitted by the email of October 30, 2003, was received and reviewed by me or my 
co-inventor. Further, from my personal knowledge the enclosed invention disclosure and the 
enclosed draft application are authentic documents that were kept in the business records of the 
company and/or the law firm in the ordinary course of business. 

I hereby declare that all statements made of my own knowledge are true, and that all 
statements made on information and belief are believed to be true; and further that these 
statements were made with the knowledge that willful false statements and the like so made are 
punishable by fine or imprisonment, or both, under Section 1001 of Title 18 of the United States 
Code, and may jeopardize the validity of the application or any patent issuing thereon. 
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, Proposal Title . 

A Methpd for-Hierarohloal Data Clustering 

Brief Description ' • 

We present a msiho'dfor hierarchical data clustering which can be applied to a variety of. 
datatypes. The approach Is unsupervised and makes minimal assumptions regarding the 
structure or statistics of the data. We have applied the technique to 'event clustering of ' / 
. digital photographs by their timestamps, and present results and comparisons to • 
competing approaches. . 
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1 Introduction • • 

This invention is In the field of media analysis. We present a method for hierarchical struc- 
tural segmentation of media streams, applicable to audio, video, and text. The system has 
been developed for hierarchical clustering of digital photographs into time-based events. 
As digital cameras proliferate, the number of digital photographs accumulating on personal 
computers is growing rapidly. As a result, there Is a demand for automatic tools to aid in 
'the organization of these collections. 

DIgitaPcameras commonly record a wealth of meta-data Into the resulting digital image 
flies. This data is typically stored in a standard Exchangeable image file format (Exir) [1] 
image header as part of each JPEG image file. This data includes the timestamp, namely 
the time the digital photograph was taken, though this Information may not be preserved 
If trie image file is subsequently modified, In a recent report, Stanford researchers have 
found that organizing photos by time significantly Improves users' performance In a se- 
ries of retrieval tasks .[2]. Furthermore, consumers often wish to organize, their photos in • 
terms of "events" both for browsing and retrieval In their own collections, as well as for 

-•sharing selected subsets of photos With others. Events are difficult to define quantitatively 
ar consistently, but most commonly, photographs ascribed to the'samB event were, taken In 
relatively olo'se proximity in time/Events tend to be difficult to describe Interims of low-level . 
Image features, and it Is easy to Imagine visually dissimilar photos belonging' to the 'same ' 
event. For example, pictures from a trip to the beach could Include photos, at different, 

. times of day, of the sky, beach or ocean, but also of various people, 

In'such contexts, content-based Image similarity may be less useful for photo clustering 
or event detection than meta-data, In particular, we focus here on processing the photos' 
timestamps, In the future, many consumer digital cameras will also record GPS location 

. information which should also prove valuable for this task. As a result, we 'have designed 
our framework to be flexible to expand.ln the future to include other meta-data and cdnteht- 
'based 'image features which may prove useful. We also n.ote that there Is a, wealth of 
digital file, types with timestamps, mainly'tlrrie of modification as stored by operating system 
software. These can also be processed similarly to. the digital image flies. As well, data 
from system or server logs can be mined for structure using time information using these . 
methods.; ... • ' ' . '• . 



2 Technical Details 

We illustrate the approach by clustering 512 photos supplied by Fujifllm Software, 'Califor- 
nia (FSCA). Al! the photos had timestamps and were placed into meaningful folders by the 
photographer. We process only the timestamps of the photos, and compare our results to 
the ground truth olustering provided by the folders in which each Image was placed. We 
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assume that Images In the same folder are part of the same event. . 

2.1 Pre-processing 

Exrf headers are processed to extract the timestamp. It should be noted that if the image is 
modified In several common Image viewers and subsequently saved, the Exif Information 
may be lost. In this case, we must rely on the modification time of the digital Image file, 
which will generally be far less reliable for event detection, the first step Is to order the 
photos In time. Denote the number of photps processed as JV. If the resulting tlmestamps 
are {** : i = 1, • • ■ ,*iV}> *en 

h<'h <•••<% * 

For now, we use minutes as the basic time unit. Note that throughout, we index matrices 
and novelty-scores by item Index (I.e. by time order) and not by absolute time.. This is 
a key' difference from previous Inventions based on similarity analysls,-the time difference 
between indices (photos) is non-uniform. In fact, time Is the (scalar) feature measured for 
each Image, as opposed to more common content-based (vector) features such as color 

• histograms'. . • , 

2.2 Distance Matrix Embedding 

To assess'structure In the collection of tlmestamps, we build a similarity matrix. As an ' 
example, the left, panel of Figure 1 shows the similarity matrix generated from the ground 
■ truth clustering. 'The elements of the matrix are one for photos from the same folder and 
zero- otherwise. The photos are indexed- In time order, such that the element of the' 

• matrix compares the names of the folders In which i* ft and }" tft photos were stored. The 
blocks along the main diagonal of the matrix- are the groups of photos in .each folder, A 

' checkerboard pattern along the main diagonal Indloates the boundary 'between folders or 
events. The crux of the checkerboard pattern is the boundary in time order between the 
photos In the two events. This is a convenient visualization of the data, which Immediately • 
shows-thatthe ground truth event clustering partitions the photos contiguously in time (I.e. 
events are contiguous and disjoint). For event clustering, we. build a family of similarity 
matrices. Indexed .by the number, of .photographs according to the general formula ' 



The parameter K controls the sensitivity of the exponential similarity measure. K has the 
unit minutes (trie similarity measure is unitless). By varying K, we can visualize the simi- 
larity between the tlmestamps at differing granularities. The left column of Figure 2 shows 
several similarity matrtices computed uslng'(1 ) for different values of K. As expected, the 
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matrices for larger values of K Illustrate ooarser olusterings of the photos' tlmestamps. 
For smaller K thB finer dissimilarities between groups of timestamps become apparent. 
The similarity measure can be tailored to Integrate or emphasize other features 'such as ' 
low-level image features, GPS data, or other meta-data! 



2.3 Segmentation 

' To cluster the photos hierarchically, we compute photo-indexed novelty scores following 
[3, 4] for each 5%. We take a matched filter approach, correlating a kernel along the main 

'diagonal of Sk- To find boundaries between groups of photos with similar tlmestamps, we 
use a Gaussian-tapered 11 x 11 checkerboard kernel, denoted g and calculate the score 



(2) 



For Implementation, there is no need to compute the full similarity matrix. _ Rather, we 
need only compute the strip around the' main diagonal with the same width as the- kernel. . 
This reduces computational-complexity to become approximately linear with the number of •' 
photos/iV, , • ' • , . 

. The right panel of Figure 1 shows the 'novelty score computed from the ground truth 
, similarity matrix using g. Figure 2 shows several similarity matrices and the corresponding, 
novelty scores computed for if = 10 3 ,10*10 B minutes. The matrices reveal structure -at 
different resolutions, In turn, the peaks in the corresponding novelty scores indicate a' 
hierarchical set of cluster boundaries between contiguous groups of photos with similar 
timestamps. In the plots of the right c'oiumn, the ground truth novelty score of Figure 1 is 
superimposed In blue. ' _ ' ' . 




Figure 1; The left panel shows the ground truth similarity matrix. The right panel shows 
the novelty score computed using a gaussian checkerboard kernel. 
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Figure 2: The left panels show the similarity matrices S# for K = 10 s (top row), 3,0* 
(middle row), and K = 10 B (bottom row). The right panels show the corresponding novelty 
. scores computed using a gaussian checkerboard kernel in green, with the ground truth 
novelty score in blue, ■ ' . 



2.4 Clustering via Boundary Detection' 

To assess performance we locate peaks [n the novelty score at each scale, enforcing a 
hierarchical structure on the detected boundaries. We perform the. analysis from coarse 
scale to fine (decreasing K). To build a hierarchical set of event boundaries, we Include 
the event boundaries detected at coarse scales in the boundary lists for all finer scales. 
'Additional study of the novelty score helps to motivate this approach. Figure 3 shows the 
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novelty scores evaluated at the eaoh of the ground truth cluster boundaries, as they vary 
with K. Clearly, the novelty scores at these boundary points vary considerably depending 
on the scale of the analysis. This is produced by the fact that the photos in the clusters on 
either side of the boundary exhibit different within -class similarity for different values of if 
per (1), This in turn varies the strength of the correlation with the checkerboard kernel, 

The maximum possible novelty score Is determined by the maximum of the similarity 
measure (one, here) and the kernel correlated along the main diagonal of the similarity 
matrix. For the 11 x 11 tapered checkerboard kernel employed here, the maximum novelty 
is 40. Intuitively, Figure 3 shows that different boundaries peak In novelty at different scales. 
In our application, this stems from the fact that different events have different time extents. 
To better illustrate this point, Figure 4 shows the boundary at photo index 285 as it appears 
in two different similarity matrices. Thus different boundaries will be dBteoted for particular 
values of # In (1). 



' Figure 3: The figure shows the novelty scores computed forthe ground truth cluster bound- 
aries varying with K In (1). 

Our approach Is based on the assumption that detected event boundaries must, at 
some scale (i.e. for some K)-, approach 'maximum novelty.. For each scale K, ws detect 
peaks iri the novelty score. by analysis of its first difference. Each peak must excBed'a 
'threshold of 20. We avoid spurious peaks by using a minimum threshold^., Specifically! we, 
select one peak In eaoh .contiguous region of the novelty score which Is greater than 5. the 
minima In the score correspond to regions of high self-similarity In S (i.e. low novelty). Thus 
'we preferentially locate boundaries between' regions of high self-similarity. We analyze the 
.matrices ordered by decreasing value of K, and impose a hierarchical structure on the 

1 Both the minimum and maximum thresholds should be determined according to the kernel employed, as . ■ 
described earlier. . 
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' detected event boundaries. 




'Figure 4: The left panel shows an excerpt from the similarity, matrix computed for K - 10. 
The right panel shows shows an excerpt from the similarity matrix computed for K ='1000. 
The variability in the {similarity within the clusters around the' boundary at 28,5 produce the 
. variations in the novelty score shown in Figure 3. 



2,5 Selecting a level of the hierarchy 

This approach hierarchically characterizes structure in'the'photo collection at multiple res- 

• olutlons. To present the results of the event clustering 'to users, we automatically rank 
each level in the Hierarchy of detected boundaries. For this we employ a confidence, mea- 
sure based on the average within-class similarity and the between-class dissimilarity of the 

• Imagery. Denote the detected boundaries at each levei/ftr = {&!,■•• , Again,, the 
boundaries, like trie-similarity matrices, are Indexed by photo not by time. For convenience, 
assume that to. and b nic -are the Indices of the first (1) and last photos (JV)> respectively. We 
then, compute the confidence score . ■ 



0{B K ) 



ii+i 

w*l •• •• . ■ 



The terms in the first sum above quantify the average wlthln-ciass similarity between the . 
. images within each cluster. The second sum quantifies the average between-class similar- 
ity between Images in adjacent clusters.. By negating this sum In the score, we quantify the 
between-cluster dissimilarity. The confidence measure combines the sum of each duster's 
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average similarity and the sum of the dissimilarity between adjaoent'dusters. Figure 5 
Illustrates the idea graphically. The wlthln-class similarity terms are the averages over the 
terms of darker regions along the main diagonal. The between olass terms are the average 
of the gray regions off the main diagonal. 




. Figure 5:. The figure Illustrates the computation of the confidence score. 'The dark re- 
gions represent within-cluster similarity, while the gray regions represent between-cluster 
similarity.' * . • • 

This confidence measure -depends on both the number of detected clusters and K 
explicitly. Figure 6 Illustrates Its behavior. The three panels.show the regions' of the re- • 
spective similarity matrices averaged and summed to form the confidence-.measure of (3) 
for K = 1778.28 .(top left), K = 100o'(top right), and K - 662.34 (bottom). For visualization 
purposes, we have set the remainder of the matrices to zero {elements not contributing to 
0(Bk)). .For K » 1778,28, we expect a lower confidence score than for isf =* 1000, since; 
the upper right panel shows fewer clusters In number and clustered regions of relatively 
low slmilarity.'"0n the 'other hand, the matrix 'for K = 562.34 shows more clusters than 
for K - looo',' but 'because K is smaller, again regions of jpw similarity are' clustered. In 
• this manner, the confidence measure emphasizes the most appropriate scale for similarity • 
analysis, The algorithm Is summarized below. 
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Algorithm 1 [Hierarchical Event Detection] • 

/. Sort photo's tlmestamps from first to last {v ,tn},U<tj &i<j- 
2. For eaoh Kin decreasing order. 

(a) Compute the similarity matrix 8k using (1). (Only part of the matrix need be ' 
computed). 

(b) Compute the novelty score vk of (2). 

(c) Detect peaks In the novelty score. 

(d) Form event boundary list using event boundaries from previous iterations and 
newly detected peaks. • 

■ 3. Compute confidence score using list of event boundaries, Bk computed for eaohK 
following (3), • " 
4. Select event boundary list for K maximizing the confidence score. • 

2.6 Experimental Results ' • : 

The precision and recall results for experiments with ,1036 Images from the F8CA data set 
appear in Table 1 In the Appendix. The exponential family of similarity measures sweep 
out a broad range In. the precision and recall of the detected cluster boundaries, "The 
confidence score is shown versus the resolution parameter K in Figure 7. • _ 

.For additional testing, we compare our approach to two other approaches, described 
in the literature % 5] and to a simple (constant) threshold. The first two algorithms are 
closely related. 'They first sort the photographs In time order and compare the time dif- 
ference between adjacent photographs to an adaptlvely computed threshold based on the 
logarithm of the 'average inter-photo time difference over's local window of photos. When 
the time difference exceeds the threshold, an event boundary Is labelled between the two. ■ 
photos. For testing, we applied the four algorithms to a collection of 1 036 photos taken 
over 15 months 2 . The photos had time stamps and were placed into meaningful folders -by ' 
, the photographer. We used those folders as ground truth for event boundaries (67 event . 
boundaries). For each algorithnvwe measured .the preolslon and recall for the detected 
event boundaries: . . . . 

_ Number of correctly detected boundaries • ... 

Precision - . Total numbeF of deteotec | boundaries ' w 
Number of correctly detected boundaries ' .„ 
Recall - Total number of true boundaries- • 



2 The 512 photos used to generate the Figures earlier In the paper were the first 512 photos taken In the 
' set of 1036 photos used for testing and comparison against othar methods. ' ■ 
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Figure 6: The three panels show'the regions of the respective'simllarlty matrloes averaged 
and summed to form the confidence measure of (3). The top row shows the matrix for K = 
1778.28 (left) and K « 1000 (right). The bottom panel shows the- matrix for J5T'«= 562.34. 



and also calculated the F-score a 



_ 2 (Precision x Recall) 
•Precision + Recall 



(6) 

The algorithm described In [5J found all event boundaries but also detected many ad- 
ditional ones (precision 0.39, recall 1,0; F-score 0.564, o' missed, 88 false positives). Be- 
cause we wanted to see whether the algorithm described in [2] could produce better results 
than simple thresholding, we skipped its thresholding step and' looked at the first .level of 
the hierarchy it created. At that level, it produced similar results as the algorithm described 
in [5] (precision 0.38, recall 1 .0; F-scdre 0.548, 0 missed, 94'false positives). 

The.seif-slmllarlty and the simple threshold algorithms have a maximum F-score when 
both the threshold and K- equal 688.33 minutes (11.5 hours). At that value, the self- 
similarity algorithm has an F-score of 0.833 (precision' 0.79, recall 0.88, 7 missed, 13 
false .positives). The simple threshold algorithm has an F-score of 0.818 (precision 0.72, 
recall 0'.95, 3 missed, 21 false positives). -At most time intervals, the self-similarity aigo- 
• rithm performs slightly better than the simple threshold algorithm. Also, the self-similarity 
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. Figure 7: The plot shows the confidence score versus lo^{K) for the 1 038 photo FSCA 
data. The corresponding precision and recall results appear in Table t, 

algorithm does not require that the threshold be determined a priori Furthermore, it sup- 
plies a hierarchy of event'bouhdarles. This Is desirable as different users will have different 
preferences forths time extent of the events they define within" their collections. Our ourrent 
Interface.allows users to explore other levels of organization in the hierarchy. 
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6 Appendix 



Table 1 : Table summarizing precision and recall for detected peaks at multiple scales. Test 
data Is 1036 images from the FSCA data, Trie row In bold indicates the resolution which 
maximizes the confidence score'. 



K 


Precision 


Recall 


F-score | Confidence Score 


0,1 


0,76 
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Witnessed and Understood-by 

1 . Submitter Signature _ 

2. Submitter Signature 

3. -Submitter Signature _ 
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Communications Center ' „__,_ .^^.^..^/Eil^: 

From: Roe, Stephen-^*— — «. 

Sent: Thursda^August 07J2003 4:39 PM 

To: cooper@fxpal.com 

Cc: Kim, Seth S. 

Subject: (Archive Copy) FX/A2018 (O&B Ref. 

PRIVILEGED AND/OR CONFIDENTIAL: 

This e-mail (including any attachments) is intended only for the use of. the individual or 
entity named above and may contain privileged or confidential information. If you are not 
the intended recipient, or the employee or agent responsible to deliver it to the 
recipient, you are notified that any review, dissemination, distribution or copying of 
this e-mail is prohibited. If you have received this e-mail in error, please immediately 
notify us by e-mail, facsimile, or telephone; return the e-mail to us at the e-mail 
address below; and destroy all paper and electronic copies. 



Communications Center 

Oliff & Berridge, PLC 

277 S. Washington Street, Suite 500 

Alexandria, VA 22314 

Tel: 703-836-6400 

Pax: 703-836-2787 

Email: commcenter&olif f . com 



MESSAGE: 



Re: New U.S. Patent Application 
Inventor: Matthew COOPER et al. 
Our Ref.: 114891 
Your Ref.: FX/A2018 

Dear Matt: 

We have been assigned by PARC to prepare another new US patent application based on one of your invention 
proposals for FXPAL, in particular, for your invention titled "A method of hierarchical data clustering." We 
would like to arrange a telephone conference call at a time mutually agreeable to you and your co-inventors, 
Jonathan Foote and Andreas Girgensohn, to discuss the invention proposal, including the current status of the 
development of the invention. We expect the telephone conference to last up to an hour to discuss all aspects of 
the invention. 

Please call or email either of us to discuss the time for a telephone conference. 
Regards, 

Steve Roe and Seth Kim 



8/7/2003 





Communications Center 



Sent: 

To: 

Cc: 



From: 



M. Coopet-I^op^r^pal.com] 
Tuesday, August 12, 3003 1:25 PM 
Roe, Stejstron-JL— — ^ 
Kim,SethS.;JonFoote 

(Archive Copy) Re: FX/A2018 (O&B Ref. 114891) 




Subject: 



Hi Steve and Seth, 




time for me. 

Thanks , 
Matt Cooper 

Roe, Stephen J. wrote: 



> PRIVILEGED AND /OR CONFIDENTIAL: 

> This e-mail (including any attachments) is intended only for the use 

> of the individual or entity named above and may contain privileged or 

> confidential information. If you are not the intended recipient, or 

> the employee or agent responsible to deliver it to the intended 

> recipient, you are notified that any review, dissemination, 

> distribution or copying of this e-mail is prohibited. If you have 

> received this e-mail in error, please immediately notify us by e-mail, 

> facsimile, or telephone; return the e-mail to us at the e-mail address 

> below; and destroy all paper and electronic copies. 
> 

> Communications Center 

> Oliff & Berridge, PLC 

> 277 S. Washington Street, Suite 500 

> Alexandria, VA 22314 

> Tel: 703-836-6400 

> Fax: 703-836-2787 

> Email: commcenter@oliff.com <mailto:commcenter@oliff.com> 




> Re: New U.S. Patent Application ■ 

> Inventor: Matthew COOPER et al. 

> Our Ref . : 114891 

> Your Ref. : FX/A2018 
> 

> Dear Matt: 
> 

> We have been assigned by PARC to prepare another new US patent 

> application based on one of your invention proposals for FXPAL, in 

> particular, for your invention titled "A method of hierarchical data 

> clustering. " We would like to arrange a telephone conference call at 

> a time mutually agreeable to you and your co- inventors,- Jonathan Foote 

> and Andreas Girgensohn, to discuss the invention proposal, including 

> the current status of the development of the invention. We expect the 

> telephone conference to last up to an hour to discuss all aspects of 

> the invention. 
> 

> Please call or email either of us to discuss the time for a telephone 

> conference. 



1 



Kim, Seth S. 



From: 
Sent: 
To: 
Cc: 

Subject: 



Roe, Steph^nJ. \ 
Wednesday, August 20, 2J0( 
'Matt Coopei^ ^/ 

Kim, Seth S. 

RE:FX/A2018(O&B Ref. 114891) 



roihorrow orFriday are fine, we assume by tomorrow morning you mean . _ 
are available from 10 am- 12 noon tomorrow morning PDT (1-3 pm our time) . If that is 
convenient, please send me a confirming email, and confirm your telephone number. 

We look forward to discussing this application with you. 

Regards, 

Steve Roe and Seth Kim ^/ 

Original Messag^^j^^r 

From: Matt Coope^c^mailt^Ncoopergf xpal . com] 
Sent: Wednesday,, August 207 2003 5:21 PM 
To: Roe, Stephens^. ^/ 
Subject: Re: EX/A20 > T8-*to&B Ref. 114891) 



Hi Stephen, 



l next to schedule a phone 
conference. Tomorrow morning or friday afternoon is fine, so let me 
know what's convenient for you. 

Thanks, 
Matt Cooper 



Roe, Stephen J. wrote: 



> PRIVILEGED AND/OR CONFIDENTIAL: 
> 

> This e-mail (including any attachments) is intended only for the use 

> of the individual or entity named above and may contain privileged or 

> confidential information. If you are not the intended recipient, or 

> the employee or agent responsible to deliver it to the intended 

> recipient, you are notified that any review, dissemination, 

> distribution or copying of this e-mail is prohibited. If you have 

> received this e-mail in error, please immediately notify us by e-mail, 

> facsimile, or telephone; return the e-mail to us at the e-mail address 

> below; and destroy all paper and electronic copies. 
> 

> Communications Center 

> Oliff & Berridge, PLC 

> 277 S. Washington Street, Suite 500 

> Alexandria, VA 22314 

> Tel: 703-836-6400 

> Fax: 703-836-2787 

> Email: commcenter@oliff.com <mailto:commcenter@oliff.com> 



> MESSAGE: 

1 



Communications Center 



From: 
Sent: 
To: 

Subject: 



Matt Coopei^lcoopefwfxpal.com] 
Thursday, A ugust 21 ,£003 2:50 PM 
Roe, SteprWi J.^X 

(Archive CopyTFte: FX/A2018 (O&B Ref. 114891) 




ICMEOO louLsavakl jtf_icme.pdf MSR-tr-2002-17.pd Slaney2001.pdf stanford.pdf 
s^pdf f 

I attached several of the references. Reference [3] is issued patent 
6,542,869. 

thanks, 
Matt 



Roe, Stephen J. wrote: 



> 

> PRIVILEGED AND/OR CONFIDENTIAL: 

> This e-mail (including any attachments) is intended only for the use 

> of the individual or entity named above and -may contain privileged or 

> confidential information. If you are not the intended recipient, or 

> the employee or agent responsible to deliver it to the intended 

> recipient, you are notified that any review, dissemination, 

> distribution or copying of this e-mail is prohibited. If you have 

> received this e-mail in error, please immediately notify us by e-maxl, 

> facsimile, or telephone; return the e-mail to us at the e-mail address 

> below; and destroy all paper and electronic copies. 
> 

> Communications Center 

> Oliff & Berridge, PLC 

> 277 S. Washington Street, Suite 500 

> Alexandria, VA 22314 

> Tel: 703-836-6400 

> Fax: 703-836-2787 

> Email: commcenter@oliff.com 



> MESSAGE: 



> We will call you at 10am your time today. We look forward to going 

> over your IP with you. 
> 

> Regards., 

>■ ■ 

> Steve Roe and Seth Kim 
> 

> Original Message 

> From: Matt Cooper [mailto:cooper@fxpal.com] 

> Sent: Wednesday, August 20, 2003 6:15 PM 

> To: Roe, Stephen J. 

> Subject: Re: FX/A2018 (O&B Ref. 114891) 



> Stephen, 



1 



Kim, Seth S. 



1/ 



From: 
Sent: 
To: 
Cc: 

Subject: 



Roe, Stepl 

Tuesdaf October jA', 2003 2:16 PM 
'M. Coopf 
Kim, Seth S. 

RE: FX/A2018 (O&B Ref. 114891) 



We are finalizing the first- draft and expect to get it to you this Friday. Will that work 
with your schedule? . 



Thanks, 

Steve Roe and Seth Kim J 

Original Mes sage- *^ 

From: M. CoopeytBSlTClJ«0>TOerefxpal.cciii] 
Sent: Tuesday, (October 14]fe003 2:11 PM 
To: Roe, Stephebyjj,. — • 

Subject: Re: FX/A2018 (O&B Ref. 114891) 



Hi Steve, 




Thanks , 
Matt 



Roe, Stephen J. wrote: 



> PRIVILEGED AND /OR CONFIDENTIAL: 
> 

> This e-mail (including any attachments) is intended only for the use 

> of the individual or entity named' above and may contain privileged or 

> confidential information. If you are not the intended recipient, or 

> the employee or agent responsible to deliver it to the intended 

> recipient, you are notified that any review, dissemination, 

> distribution or copying of this e-mail is prohibited. If you have 

> received this e-mail in error, please immediately notify us by e-mail, 

> facsimile, or telephone; return the e-mail to : 'us at the e-mail address 

> below; and destroy all paper and electronic copies. 
> 

> Communications Center 

> Oli'ff & Berridge, PLC 

> 277 S. Washington Street, Suite 500 

> Alexandria, VA 22314 

> Tel: 703-836-6400 

> Fax: 703-836-2787 . 

> Email: commcenter@olif f . com 



> MESSAGE: 



1 



Kim, Seth S. 




J 



From: 
Sent: 



Cc: 

Subject: 



Roe, Steplaen J. 

Thursday/October 3(^2003 7:26 PM 
'cooper@fxpgl.cqrp - 
Kim, Seth S. 

FW: Your Ref.: FX PAL IP-02-018; Our Ref.: 1 14891 



Re: 


New U.S. Patent Application 




Inventor: Matthew COOPER et al. 




Our Ref.: 114891 




Your Ref. : FX PAL IP-02-018 



Dear Matt: 

Further to our telephone conference earlier today, we enclose a •first draft of your above-identified patent 
application in electronic form in MS Word. Further claims will be forwarded to you by tomorrow morning. 
The drawings are also being sent by facsimile to 650-813-7081 concurrently with this e-mail message. 

Steve Roe and Seth Kim 




Draft application First Draft 
letter.pdf application FX-A20... 



Communications Center 



From: 
Sent: 
To: 
Cc: 

Subject: 



Matthew Cooper [cooper@fxpal.com] 
Friday, October 31 , 2003 1 1 :35 AM 
Roe, Stephen J. 

Kim, Seth S.; sumiya@fxpal.com 

(Archive Copy) Re: FW: Your Ref.: FX PAL IP-02-018; Our Ref.: 1 14891 



n 





Thanks , 
Matt Cooper 

Roe, Stephen J. wrote: 



> PRIVILEGED AND/OR CONFIDENTIAL: 

> This e-mail (including any attachments) is intended only for the use 

> of the individual or entity named above and may contain privileged or 

> confidential information. If you are not the intended recipient, or 

> the employee or agent responsible to deliver it to the intended 

> recipient, you are notified that any review, dissemination, 

> distribution or copying of this e-mail is prohibited. If you have 

> received this e-mail in error, please immediately notify us by e-mail, 

> facsimile, or telephone; return the e-mail to us at the e-mail address 

> below; and destroy all paper and electronic copies . 
> 

> Communications Center 

> Oliff & Berridge, PLC 

> 277 S. Washington Street, Suite 500 

> Alexandria, VA 22314 

> Tel: 703-836-6400 

> Fax: 703-836-2787 

> Email: commcenter@oliff.com <mailto:commcenter@oliff.com> 



1 



Communications Center 



63 



From: 
Sent: 
To: 
Cc: 

Subject: 




heckerman96tutoria Icassp03-mtg.pdf chen98speaker.pdf 
l.pdf 



Here are references for the BIC (Bayes information criterion) . The 
first one is a general review of the BIC, Bayesian learning, and other 

David Heckerman. A tutorial on learning with bayesian networks. 
Technical Report MSR-TR-95-06, Microsoft Research, Redmond, Washington, 
1995. Revised June 96. http://citeseer.nj.nec.com/heckerman96tutorial.html 



Chen and P. Gopalakrishhan, "Speaker, environment and channel change, 
detection and clustering via the Bayesian information criterion, " in 
DARPA speech recognition workshop, 1998. 
http : // ci teseer . n j . nec . com/chen9 8speaker . html 

S. Renals and D. Ellis (2003) C Audio Information Access from Meeting 
Rooms <http : / /www . ee . Columbia . edu/ %7Edpwe/pubs /icassp03 -mtg . pdf > 
/Proc. ICASSP-03/, Hong Kong, April 2003 



thanks for your help, let me know if you need anything else. 



Matthew Cooper 

cooperSf xpal . com <mailto : cooper@f xpal . com> 



Kim, Seth S, 

From: 
Sent: 
To: 
Cc: 

Subject: 



Roe, Stephen J. 

Thursday, November 1$ 2003 1 1 :15 AM 
aj.com^, 

Kim, Seth S. 

FX/A2018 (O&B Ref. 114891) draft non-provisional application 



Re: 


New U.S. Patent Application 




Inventor: Matthew COOPER et al. 




Our Ref.: 114891 




Your Ref.: FX/A2018 



'Matt: 

We have now completed, and enclose, a draft of the non-provisional application as promised, along with the formal figures 
and a cover letter. Please review the cover letter before you review the application. 

Thanks, 

Steve Roe and Seth Kim 




Draft Application ' Draft FIGURES.pdf 
Cover Letter... i-Provislonal Applies 



Kim, Seth S. 



> / 




Please forward to: cooper@fxpal.com 



Re: 


New U.S. Patent Application 




Inventor: Matthew COOPER et'al. 




OurRef.: 114891 




Your Ref.: FX PAL JF-02-018 



Dear Matt: 



We have the following information for you and your co-inventors, Jonathan Foote and Andreas Girgensohn. 
Please corifirm their accuracy and notify us if any of the information should be updated. 

Mathew L. Cooper 

3998 23rd Street. 

San Francisco, CA 941 14 USA 

Citizenship: USA 

Andreas Girgensohn 

210 Waverly Street #6 

Menlo Park, California 94025 USA 

Citizenship: Germany 

Jonathan T. Foote 

450 Laurel Street. 

Menlo Park, CA 94025 USA 

Citizenship: USA 

Very truly, 



Steve and Seth 




'3 



Kim, Seth S. 



From: 
Sent: 
To: 

Subject: 



Roe, Steonen J. 

Tuesdayf December 02^003 3:18 PM 
Kim, SetfvS. 
FW: FX/A^etfrfO&B Ref.: 114891) 



— Original Message — 
From: Roe, Stephen J. 

Sent: Tuesday, November 25, 2003 4:49 PM 

To: 'cooper@fxpal.com' 
Subjects FX/A2018 (O&B Ref.:. 114891) 



Re: 


New U.S. Patent Application 




Inventor: Matthew COOPER et al. 




Our Ref.: 114891 




YourRef.:RX/A2018 



Dear Matt: 

Thank you for your November 24 e-mail messages. We have attached clean and marked-up versions of the final 
draft of your above-identified patent application and the formal drawings, Figs. 1-18, for you and your co- 
inventors,' review, along with the Declaration/Power of Attorney and the Assignment. 1 



Please call either of us if you have any questions. 



Steve Roe and Seth Kim 



Final Draft 
Applicatlon.zip 



